A time-varying image sequence is a sequence of images of a given scene with each successive image taken some time interval apart from the one preceding it. The velocity vector field of an image is defined as the collection of the two-dimensional velocity vectors which are the projections of the three-dimensional velocity vectors of the visible points in the scene. If the scene being imaged changes gradually with time, and if the changes are mostly due to the relative movements of the physical objects in space, then the corresponding changes in the successive images of the sequence can be used to estimate the velocity vector fields of the images.
Reliable estimation of velocity vector fields is very important for the analysis of time-varying image sequences. The temporal variations in the images of the sequence specified by the velocity vector fields can be used to extract spatial information about the scene. They can also be used to extract information characterizing the movements of the objects in the scene. Stereopsis can be considered as a special case of image-sequence analysis where the number of images is restricted to two and the displacement direction is known.
There are two principal approaches to the problem of estimation of the velocity vector field: the feature-based matching approach and the spatio-temporal gradient approach. For feature-based matching approach, the image points with significant variations in the values of the time-varying image function, called feature points, are identified in both images. The estimation of the velocity vector field is accomplished by matching the feature points of one image to the feature points of the other image. The spatio-temporal gradient approach is based on the constraint imposed on each velocity vector relating the spatial gradient of time-varying image function to the temporal derivative of the time-varying image function.
The spatial variation in the time-varying image function utilized in the above approaches do not provide sufficient information to determine the estimate of the velocity vector field. In the feature-based matching approach the velocity vectors can be estimated only on a sparse set of image points, while in the spatio-temporal gradient approach at most one constraint is imposed on two components of each velocity vector. To overcome these difficulties, it has been proposed, Horn and Schunk, "Determining Optical Flow", Artif. Intell. 17, 1981, pp. 185-203, that velocity vector fields should vary smoothly from point to point on the image plane. This requirement enabled estimation of both components of the velocity vector at each image point; however, it forced the estimate of the velocity vector field to vary smoothly across the occluding boundaries. Several approaches, which are based on the selective application of the smoothness requirement, have been proposed to overcome this difficulty.
Yachida used estimates of the velocity vectors at prominent feature points as reliable initial estimates and sequentially propagated them into neighboring image points. Davis et al first computed estimates of the velocity vectors at corner points and then propagated them along the contour lines between corner points. Hildreth estimated the velocity vector field along zero-crossing contours. Nagel used image points with significant second-order spatial variation to estimate of velocity vector fields. Nagel and Enkelmann investigated oriented smoothness constraints for the estimation of velocity vector fields. Terzopoulos introduced controlled continuity measures to overcome the discontinuities in visual problems. Hierarchical approaches to the problem of estimation of velocity vector fields have also been investigated.
There is a need for a method of modifying a time-varying image sequence based on an improved method of estimating the velocity vector field.